Most video applications seek the highest possible perceptual quality for a given set of bit rate constraints. For example, in a low bit rate application such as a videophone system, a video encoder may provide higher quality by eliminating the strong visual artifacts at the regions of interest that are visually more noticeable and therefore more important. On the other hand, in a high bit rate application, visually lossless quality is expected everywhere in the pictures and a video encoder should also achieve transparent quality. One challenge in obtaining transparent visual quality in high bit rate applications is to preserve details, especially at smooth regions where loss of details are more visible than that at the non-smooth regions because of the texture masking property of the human visual system.
Increasing the available bit rate is one of the most straightforward approaches to improving objective and subjective quality. When the bit rate is given, an encoder manipulates its bit allocation module to spend the available bits where the most visual quality improvement can be obtained. In non-real-time applications such as digital video disk (DVD) authoring, the video encoder can facilitate a variable-bit-rate (VBR) design to produce a video with a constant quality on both difficult and easily encoded contents over time. In such applications, the available bits are appropriately distributed over the different video segments to obtain a constant quality. In contrast, a constant-bit-rate (CBR) system assigns the same number of bits to an interval of one or more pictures despite their encoding difficulty and produces visual quality that varies with the video content. For both variable-bit-rate and constant-bit-rate encoding systems, an encoder can allocate bits according to perceptual models within a picture. One characteristic of human perception is texture masking, which explains why human eyes are more sensitive to loss of quality at the smooth regions than in textured regions. This property can be utilized to increase the number of bits allocated to the smooth regions to obtain a higher visual quality.
The quantization process in a video encoder controls the number of encoded bits and the quality. It is common to adjust the quality through adjusting the quantization parameters (QPs). The quantization parameters may include the quantization step size, rounding offset, and scaling matrix. In the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Moving Picture Experts Group-4 (MPEG-4) Part 10 Advanced Video Coding (AVC) Standard/International Telecommunication Union, Telecommunication Sector (ITU-T) H.264 Recommendation (hereinafter the “MPEG-4 AVC Standard”), the quantization parameter values can be adjusted on a slice or macroblock (MB) level. The encoder has the flexibility to tune quantization parameters and signal the adjustment to the decoder. The quantization parameter signaling requires an overhead cost.